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Abstract 

These proceedings present the unfolding techniques used so far in ATLAS. 
Two representative examples are discussed in detail; one using bin-by-bin cor- 
rection factors, and the other iterative unfolding. 



1 Introduction 

The distribution of any observable is distorted due to experimental limitations. Unfolding is the proce- 
dure of estimating the "truth-level" spectrum, i.e., the spectrum that would be measured with an ideal 
detector and infinite event statistics. A general introduction to unfolding, and details about various meth- 
ods are given in other contributions to this workshop. The focus here will be on real life examples of 
unfolding in ATLAS analyses. 

fN| \ As of early 2011, ATLAS has used two unfolding methods: 

Q_i| i) bin-by-bin correction factors; 

\ ii) the iterative method by D'Agostini HI. 

in 

One representative example will be presented from each method. In Section |2J bin-by-bin correction 
is presented through the inclusive jet px spectrum measurement [2 |. In Section |3l D' Agostini's itera- 
>^ \ tive method [T] is presented, as it was used to estimate the spectrum of charged particle multiplicity in 

^ ' minimum bias interactions 0. 

Both methods have drawbacks. An insightful overview can be found in J4]], and in other contri- 
butions to this workshop. Bin-by-bin correction has been particularly criticized for not dealing carefully 
with bin correlations, among other things. ATLAS is considering methods beyond bin-by-bin in the next 
round of analyses where this method was used. 

In searches for new physics, ATLAS does not apply any unfolding, because it is unnecessary for 
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\Q • making a discovery, or for setting a limit to some model, or for estimating model parameters. Unfolding 

can be regarded as useful when the distribution itself (or a binned version thereof) is regarded as the set 
of parameters of interest. 

o 

2 Bin-by-bin correction factors 

Several ATLAS analyses have used the method of bin-by-bin correction factors ||2]|5]-|7], mostly because 
of its simplicity. The example of inclusive jet px measurement Q will be discussed. The main result of 
this measurement is shown in Fig.Q] In this analysis truth-level corresponds to hadron-level. 

2.1 Method description 

Let Ti be the expected number of events in bin i of the truth-level px spectrum, which is obtained from 
Monte Carlo (MC). Leading order Pythia 00 QCD MC was used in the case of O, where no event 
selection was applied. The truth-level px spectrum contains jets reconstructed after hadronization, apply- 
ing the anti-ZcT clustering algorithm on stable hadrons produced after fragmentation and hadronization. 
Detector simulation is not involved in the truth-level spectrum. 

Let Ri be the expected number of events in bin i of the measured px spectrum, after event selection 
which includes trigger requirements, jet reconstruction inefficiency at low px, primary vertex require- 
ments, jet quality criteria etc. The same Pythia QCD MC is used as before, after ATLAS detector 
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Fig. 1: The estimated truth-level spectrum of inclusive jet pt (filled markers) from J2), obtained using bin-by- 
bin correction factors, compared to the theoretical truth-level QCD prediction (red band). The black error bars 
represent the statistical uncertainty of the estimated spectrum, and the blue band the total systematic uncertainty, 
which is obtained by summing in quadrature individual systematic uncertainties. The dominant contribution comes 
from the jet energy scale uncertainty. In each bin the estimated truth-level spectrum has been divided by the width 
of the bin and by the integrated luminosity, whose uncertainty (1 1%) is not included in the blue error band. 



simulation, to obtain Ri. Jets are reconstructed by applying the same anti-A^ algorithm on topological 
clusters of energy deposited in the calorimeter [|9]. 

Let D{ be the actually observed number of events in bin i of the measured px spectrum. Whereas 
Ti and Ri are both real numbers after normalizing the MC samples to the integrated luminosity of the 
available dataset, Di can only take integer values, because the observed events are discrete. If it is 
assumed that Ri is the result of an ideal simulation of all physical processes that occur at the proton 
collisions 1 and of the ATLAS detector, then Di is a random integer that follows a Poisson distribution 
with mean R, L . 

Let 

Ci = ^, (1) 
tti 

be the correction factor corresponding to bin i of the observed px spectrum. The correction factors used 
in O are shown in Fig. |2] 

'Obviously this is not a good assumption when one acknowledges the possibility of new physics, but in measurements such 
as the one we discuss here it is presumed that what is measured is just QCD. 
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Fig. 2: The correction factors (Cj) used in 0. The statistical uncertainties (black crosses) are invisibly small. The 
green band represents the total systematic uncertainty, except for the part which is due to jet energy scale, which 
is discussed in Section l2.3.4l 



The answer returned for bin i of the truth-level pt spectrum after bin-by -bin correction is 

Ui = d- A- (2) 

Ui is the estimator of T, . 



2.1.1 Bias 

The estimator U{ has a bias that is easy to compute. 

Let's consider the possibility that the truth-level spectrum is actually T[, which may differ from the 
assumed Tj. This could happen, for example, if sizable processes other than those included in PYTHIA 
QCD are occurring in nature, or if the modeling of QCD by PYTHIA is unrealistic. Let's also assume 
that the actual expected spectrum at detector level is R[, which may differ from R4 for the above reasons, 
as well as due to unrealistic modeling of the detector response and of the quantities involved in event 
selection. The bias of the estimator Ui then is 



(Ui - r/> = (|-a - it) = -n= §^ 



r (§-|u;. 



(3) 
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Fig. 3: Sketch of the Neyman construction used to correspond an observed number of data events Di to a 68% 
confidence interval for Ti in the bin-by-bin correction factor method. The definitions of Di, Ti, R4, Ci and Ui are 
given in Sec. 12. II 

2.2 Statistical uncertainty 

The Neyman construction shown in Fig.|3]is effectively used to obtain a confidence interval for Ti, given 
Ci and the data Di. Having observed Di, the 68% confidence interval (CI) for Ui is approximately 



This is a fair approximation when Di is large, in which case the Poisson distribution of Di with mean 
Ri is similar to a Gaussian of mean Ri and standard deviation \fRl. Although this approximation fails 
in bins with few data, the same formula was used in all p? bins, so for all bins it was assumed that the 
statistical uncertainty of Ui is symmetric and equal to 



This is the size of the black error bars in Fig. Q] 
2.3 Systematic uncertainty 

The following main sources of systematic uncertainty were identified in : 

i) the correction factor Cj is subject to statistical fluctuations due to finite MC event statistics; 

ii) the amount of pr smearing in detector simulation may be unrealistic; 

iii) the used spectrum of Tj may be unrealistic; 

iv) jet energy scale uncertainty. 

The following paragraphs describe how each systematic uncertainty was propagated to the final 
estimator of the truth-level spectrum. 
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Fig. 4: The effect of co variance between T,; and Ri in the variance of the correction factor Cj. The gray circle 
indicates the case of zero covariance, and the blue circle the case of highly positive correlation. 

2.3.1 Finite MC event statistics 

The correction factors C{ have some uncertainty due to random fluctuations of the finite MC event statis- 
tics available to determine Tj and Ri. When Tj fluctuates above its mean, so does Ri, so the two are 
highly correlated (Fig. |U. 2 The statistical uncertainty in Q was computed taking this correlation into 
account, as follows: 

The N(Ri) MC evens which compose Ri are separated into those that are coming from the same 
truth-level bin (N(Ri A T)) and those originating from different truth-level bins (N(Ri A -Tj)). 3 Sim- 
ilarly, the N(Ti) MC events which contribute to T are separated into those that end up in the same 
bin after detector simulation and event selection (N(Ti A Ri)), and those that migrate to different bins 
(N(Ti A -iRi)). The variables N(Ti A R4) and N(Ri A T) are identical. So, Cj can be expressed as a 
function of three statistically independent random variables: 

c = Tj_ = N(Tj A Rj) + N(Tj A -nflQ 
i Ri N(Ti A Ri) + N(Ri A -fl}) " 

Since Cj is expressed as a function of three statistically uncorrected variables, error propagation can be 
used where covariance terms are zero. Each one of the three MC event populations has standard deviation 

Vn. 

2.3.2 Jet pt resolution uncertainty 

A relative systematic uncertainty of 15% in jet px resolution was assumed, based on the results of in-situ 
studies ifTOl . 

To model the effect of a different px resolution on d, the jets in MC events were smeared by an 
additional amount a, which varied from to 20% of the nominal smearing that is present in ATLAS MC. 
For each amount of extra smearing, the values of Ri change, while T, is not affected. As a result each 
correction factor C{ has a dependence on the amount of extra smearing. It was furthermore observed that 
in all bins i the correction factor C\ increased about linearly with a. 

It is possible to increase the smearing of jet pt by adding to it a random offset of appropriate 
variance, but it is not possible to do the opposite, i.e., to reduce the amount of smearing that is nominally 
present in the ATLAS MC. This complicates the task of determining the uncertainty on Cj, because the 
resolution uncertainty of 15% is symmetric; the jet px resolution could be 15% worse or 15% better than 

2 Though this was not done in the analysis described here, it may be worth mentioning that T, and R4 could be generated 
separately, using different seeds for the pseudo-random numbers in the MC generator. This would result in statistically inde- 
pendent estimators Ti and Ri . In addition to that, since T, does not involve detector simulation, it is feasible to generate many 
more truth-level MC events, thus estimating Ti with negligible statistical uncertainty, something that unfortunately is not easy 
for R4. 

3 The symbol A is the logical "and", while -1 is the logical "not". So, Ri A ->Ti means belonging in Ri and not in Ti. 
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its nominal value. The observation that Cj depends linearly on the extra smearing justifies the assumption 
that, if the resolution improved, C, would decrease, linearly, at the same rate. 

Therefore, the systematic uncertainty on Cj due to 15% uncertainty on jet pt resolution is de- 
termined by noting the increase of Cj when resolution is made 15% worse, and by symmetrizing this 
variation. For example, if Ci changes by +1% when jet px resolution is deteriorated by 15%, we assign 
to C{ a systematic uncertainty of ±1%. 

2.3.3 Uncertainty in spectrum shape 

The correction factors C depend on the choice of Tj, which affects also Ri. If, for example, Pythia 
QCD does not provide a realistic model of the true spectrum, that can bias Ui, unless Ri and Tj are 
simultaneously wrong in such a way that Ti/Ri remains equal to the (unknown) actual ratio T-fR^ in 
Eq. [3] The use of bins quite wider than the amount of smearing makes it more likely that, even if Tj is not 
modeled right, the ratio Z$ / Ri in each bin will be approximately correct. In [2j the bins are safely wider 
than jet px resolution, and their edges are driven by experimental constraints, such as trigger thresholds. 

To assess the uncertainty from possible wrong modeling of Tj, the MC events used to determine 
Ci were re-weighted in multiple ways. Their re-weighting was determined by functions smooth in jet 
Pt, chosen so as to bracket the variation observed by varying partem density functions, by including 
next-to-leading-order corrections to QCD, as well as the difference observed between Di and R4. For 
each set of re-weighted MC events both Tj and R4 were re-computed, and so was Ci for each bin i. The 
largest variation observed in each Cj was taken as a systematic uncertainty. 

2.3.4 Jet energy scale uncertainty 

By far the dominant uncertainty in the final {7, comes from the uncertainty in jet energy scale (JES). All 
previous uncertainties, added in quadrature, amount to about 5% of relative uncertainty in Cj, which is 
the error band shown in Fig. |2] The rest ~40% of uncertainty in the final answer comes from the JES 
uncertainty, and it dominates the blue error band in Fig. [T] 

To propagate the JES uncertainty, the reconstructed px of all jets in MC events is shifted by ±1 
standard deviation, the exact size of which is a function of jet px and pseudo-rapidity 77. That affects Ri 
strongly, while Tj doesn't change, therefore Ci varies significantly. By applying on Di the two alternative 
values of Ci, from the positive and the negative JES shift, two extreme Ui values are obtained for each 
bin i, whose distance is considered as the JES uncertainty on Ui. 

3 Iterative unfolding 

ATLAS used D' Agostini's iterative unfolding [1] in the study of minimum bias pp collisions 0. The ex- 
ample to be shown is the estimation of the truth-level distribution of the multiplicity of charged particles. 
The result of this analysis is shown in Fig. [5] 

3.1 Method description 

The full method is clearly described in the original article [1] by D' Agostini. This paragraph will make 
a connection between the quantities in and the notation used in flX 

Let n c /j be the number of charged particles produced in a pp collision. This is the truth-level 
quantity whose distribution needs to be estimated. It corresponds to the "cause" C mentioned in (T). 

Let n tr k be the number of reconstructed tracks in a pp collision, which satisfy the selection criteria 
listed in [3 ]. It corresponds to the "effect" E mentioned in (T). 

The reconstructed tracks are typically fewer than the actual charged particles, due to tracking 
inefficiency, therefore typically n tr k < rich- Therefore the migrations matrix is highly non-diagonal, and 
schematically looks like Fig. [6] 
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Fig. 5: The estimated spectrum of charged particle multiplicity (filled circles) in minimum bias pp interactions, 
from 0. The statistical uncertainty is smaller than the marker size, and the asymmetric color band represents the 
total systematic uncertainty. Various theoretical predictions are overlaid for comparison. 
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Fig. 6: Schematic representation of migrations matrix. The dark green squares represent higher probability than the 
light blue. Initially each matrix element equals the probability of MC events to contain n c h charged particles and to 
have n t rk reconstructed tracks that satisfy the criteria listed in flT). Then, the elements of each row, corresponding 
to a fixed n tr k, are normalized to have sum 1. 
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Re- writing the basic formulas from [1 1, substituting C — > n c h and E — > n trk , we g et 

N(n ch ) = 1 ^ N{n trk )P{n ch \n trk ) e(n ch ) / 0, (7) 
e[Jlch > n trk >2 

r{n ch \n trk ) - ^ — ; — r, (8) 

where the efficiency e(n c / l ) corresponds to the probability of reconstructing at least two tracks, a require- 
ment related to having a reliable primary vertex reconstruction, for a given number of charged particles: 

e(n ch ) = P(n trk > 2\n ch ). (9) 

The term Po(n c h) is an arbitrary initial distribution for the truth-level quantity n c h. The symbol N(nt rk ) 
denotes the population of events where n trk tracks were reconstructed, and N(n c h) is the estimator of 
the population of events with n c /, charged particles at truth-level. 



3.1.1 Initial distribution and iterations 

In O, the initial distribution was defined to be the n c h spectrum predicted by Pythia minimum bias 
MC. The reason is that the Pythia prediction has been tuned to data from various past experiments, so 
it is a reasonable starting point. 

In iterative unfolding the number of iterations is decided arbitrarily. Too many iterations result 
in bin-by-bin fluctuations in the unfolded spectrum, similar to what one may get from simple migration 
matrix inversion |@). Too few iterations increase too much the influence of the initial distribution on the 
final answer. 

In 0, a convergence criterion was defined to determine when to stop iterating. The criterion was 

< 1, (10) 



where 



X 2 = > ; I " cft ; " m I • (ID 




Namely, iterations continued until the latest unfolded spectrum (jf^ r ^ nt ^ remained statistically consis- 
tent with the spectrum from the previous iteration ( n P^ emous y n was found that 4 iterations were enough 
to meet this convergence criterion. 



3.1.2 The term e(n c ^) 

In principle one should extract e(n c ^) defined in Eq.|9j directly from the MC events used to populate the 
migrations matrix (Fig. |6). However, a decision was made in to use instead a parametric approxima- 
tion of e(n ch ). 

Making the simplification that each charged particle has the same "average effective" probability 
e e f f of being reconstructed as a track, the probability of having at least two reconstructed tracks is given 
by 

f(n ch ) = 1 - (1 - e eff ) n ^ - n ch (l - e e// )< n <*- 1 >e e// . (12) 

The unknown parameter e e ff was adjusted so as make /(2) equal to the e(n c h = 2) obtained from MC. 
The resulting value for e e ff is within 4% from the average probability of track reconstruction that is 
determined from MC simulation, which indicates that f{n c h) matches well the MC-driven e{n c h) even 
for n ch > 2. 
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After adjusting e e fj as described, the quantity f(n c h) from Eq. [T2l substitutes e(n c h) in Eq. [7] 
Practically this efficiency becomes ~ 1 for n c h > 4, and that is true regardless of using f{n c h) or 

3.2 Statistical uncertainty 

In Eq. |7J the estimator N(n c h) depends on the measured N{n trk ), which are the result of indepen- 
dent Poisson fluctuations. Simple error propagation would lead to the following standard deviation for 

N(n ch ): 

a N(n ch ) = \Jj2 (^-^PMn trk )^j N(n trk ). (13) 
The way statistical uncertainty was actually calculated in Q was 

a N{n ch ) = ^(n ch ). (14) 

Either way, the statistics in all bins of n tr k are high enough to make the statistical uncertainty 
negligible. In Fig. [5J the statistical error bars are invisible. 

3.3 Systematic uncertainty 

The following main sources of systematic uncertainty will be discussed: 

i) The choice of initial distribution Po(n c h); 

ii) The uncertainty in track reconstruction efficiency; 

iii) The uncertainty in MC spectrum. 



3.3.1 Choice of initial distribution 

The stability of the answer under different choices of initial distribution Po(n c h) was tested by assuming 
a "flat" initial distribution Po(n c h) = 1, and repeating the iterative unfolding procedure. This choice is 
obviously physically absurd; its purpose was only to show that even under extreme choices of Po{n c h) 
the answer N(n c h) doesn't change much. 

Starting from a flat initial distribution, the number of iterations required to converge (Eq. [10) 
increased from 4 to 7. The final answer changed by less than 2% in all bins of n c h, which was taken as a 
systematic uncertainty in N(n c h). 

3.3.2 Track reconstruction efficiency uncertainty 

The main effect this unfolding is correcting is the inefficiency of tracking. This inefficiency is reflected in 
the probabilities of Eq. |7J and is obtained from MC simulation. If tracking inefficiency in MC is wrong, 
so is the obtained spectrum after unfolding. 

Fig. |7] shows the track reconstruction efficiency (e tr k) m ATLAS simulation. 

To propagate the uncertainty of e trk into N(n c h), the natural thing to do would be to shift sys- 
tematically e^k, thus changing P(n c h\n tr k), and see how much N(n c h) would change. Instead, what 
was done in (3J was to keep the migration probabilities fixed, and modify the data (N(ch tr k)) on which 
iterative unfolding was applied. The way in which the data were modified is described next. 

Assume an event in data has n trk tracks. Take one of these tracks. Its px corresponds to some 
efficiency e^k (Fig- LZJl- For the sake of clarity, let's say it corresponds to e^-k = 0.80 ± 0.05. This e t rk 
gets reduced by 1 standard deviation, so it is brought down to 0.75. For this reduced e tr k, the expected 
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Fig. 7: Track reconstruction efficiency e t rk in ATLAS simulation. The error band represents its systematic uncer- 
tainty. 

number of tracks is x 0.75 ~ 0.94. The track is then randomly kept, with probability 0.94, or 
discarded, with probability 0.06. This procedure of efficiency reduction and random removal is repeated 
for all n tr k tracks of the event. In the end, the event is left with n' trk , where n' trk < ntrk- 

The above procedure is repeated for all data events, reducing n tr k to n' trk in each event. Then, the 
distribution N(n' trk ) is unfolded instead of N(n tr k), which results in N'(n c h) instead of N(n c h). 

The above procedure could only remove tracks, not create any. However, the e tr k uncertainty is 
symmetric, which means that the actual e^k could be also greater than its nominal value. For this reason, 
the difference between N(n c h) and N'(n c h) is symmetrized, and used as a systematic uncertainty in 
N{n c h)- That means, for example, that if in a bin of n c h the N'(n c h) was 5% greater than N(n c h), the 
uncertainty is set to ±5%. 
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3.3.3 Uncertainty due to spectrum shape 

The observed spectrum of track transverse momentum (p^ k ) disagrees with the MC prediction after full 
ATLAS detector simulation, as shown in Fig. [8] This discrepancy is related to the unfolding from ntrk 
to rich, because e tr k 1S a function of p l ^ k (Fig. [7]). If the ptf k is not realistically modeled, neither is e tr k- 

The way this was treated was the following: In each bin of n tr k, the mean e tr k was found by 
looping through all data events in the bin, and corresponding each observed ptf k to the value of e tr k 
obtained from MC (Fig.|7J. The same was then done for MC events in bins of n tr k, again corresponding 
the pt£ k in MC events to values of e tr k from Fig. [7] In each bin of n^k* the average track reconstruction 
efficiency (etrk) from data was compared to the same quantity from MC. The same p l rf k — > etrk corre- 
spondence was used for both data and MC tracks, therefore the difference in the resulting (etrk) * s due 
to the different p l f k distributions. 

In each bin of n tr k, if (etrk) is larger in data than in MC, then the efficiency in data gets reduced 
by the observed difference, in the same stochastic way described in Section I3.3.2I This results in a 
different number of tracks n' trk < n tr k for each event in the data. The iterative unfolding is then applied 
to N(n' trk ), and a different estimator of the truth-level spectrum is obtained (N'(n c h))- The difference 
between the nominal N(n c h) and N'(n c h) is found, and is used as a one-sided systematic uncertainty in 
N{n ch ). 

In n tr fc bins where the (etrk) m data is smaller than in MC, one would ideally wish to increase the 
e^k °f the data, but it is not possible to create tracks, as explained in Section I3.3.2I Instead, the e tr k °f 
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Fig. 8: The observed spectrum of p f , compared to the spectrum expected from MC simulation. 



data is reduced by the observed difference, as if the data had greater (etrk) than the MC. This results in 
a different data spectrum N(n' trk ), which after unfolding results in a different estimator N'(n c h)- The 
difference between the nominal N[n c h) and N'{n c h) is found, and instead of using it directly as a one- 
sided systematic uncertainty in N(n c h), we use its opposite, to take into account the fact that the data 
e tr k was reduced instead of increased. 

The above procedure results in an asymmetric systematic uncertainty. The upper and lower uncer- 
tainty are separately added in quadrature with the other systematic uncertainties, which are symmetric. 
This results in two unequal total systematic uncertainties, one suggesting that N{n c h) could be above 
and the other below its nominal value. This asymmetric systematic uncertainty appears as a colored band 
in Fig. [51 



4 Concluding remarks 

Two examples were shown of how unfolding has been used in ATLAS. They were chosen to be repre- 
sentative of different cases; one is using the bin-by -bin factors to correct the spectrum of a continuous 
observable (jet pr), whereas the other uses iterative unfolding to estimate the truth-level distribution of 
a discrete variable (n c h)- The systematic uncertainties are quite different, as one analysis deals mainly 
with energy smearing, and the other with tracking inefficiency. 

As of the time of this workshop, ATLAS has used extensively bin-by-bin correction factors, and 
in some cases iterative unfolding. More methods are being considered for future iterations of some of 
these analyses. 

Unfolding has been used only in analyses where the goal was to estimate a truth-level distribution. 
Unfolding has been deliberately avoided in searches for new physics, where bias in bins with low statis- 
tics can not be afforded, where it can not be assumed that the data are consistent with the MC prediction 
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as is silently assumed in some stages of unfolding, and where Poisson-distributed data are simpler to 
evaluate than estimators resulting from unfolding procedures after a series of arbitrary regularization 
choices. There are several unfolding methods, in some of which anomalies due to new physics could 
even be reduced, whereas the observed data are unique. Any inference is possible using directly the data, 
without unfolding, and a theoretical prediction that either includes full detector simulation, or at least an 
approximation of it that amounts to the inverse of unfolding, namely folding, which can be done with no 
need for regularization. The only task for which unfolding is strictly needed is the estimation of a truth- 
level spectrum, whose later use to make statistical inferences becomes complicated by non-Poissonian 
statistics, biases that are hard to estimate, and bin-to-bin correlations. 

In none of the analyses where unfolding was used was the full covariance matrix provided. The 
latter would be necessary to correctly compare a truth-level theoretical prediction to the result of unfold- 
ing. In most cases the comparison between the result of unfolding and the truth-level Standard Model 
prediction is made qualitatively, avoiding to provide a p-value that would require proper use of the co- 
variances between bins. When a more quantitative comparison is attempted, like in |6j, a \ 2 is used only 
as a metric to determine if one theory agrees with the data more than another, but not as a test statistic to 
compute a p- value. 
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